test: Fix guest_env_test.go #21801

nirs · 2025-10-25T21:34:33Z

The test used go:build iso so it was not included in the integration tests. Change to go:build integration so we test in the CI.

Fix the test for arm64, since virtualbox programs are not included in the iso.
The test is 2.45 times faster now using --no-kubernetes
Rename the file and the test name to make it more clear that this test is about the iso image.
Skip the test for non vm-driver, since this tests is about the iso image.

We may need to add a similar test or adapt this test so it can be used also with the kicbase image.

This test will be useful to validate #21800, avoiding regressions such as #21788.

Example run

% go test -v ./test/integration -run TestISO -tags integration -count 1
...
--- PASS: TestISOImage (5.77s)
    --- PASS: TestISOImage/Setup (5.30s)
    --- PASS: TestISOImage/Binaries (0.00s)
        --- PASS: TestISOImage/Binaries/wget (0.09s)
        --- PASS: TestISOImage/Binaries/socat (0.09s)
        --- PASS: TestISOImage/Binaries/iptables (0.10s)
        --- PASS: TestISOImage/Binaries/podman (0.10s)
        --- PASS: TestISOImage/Binaries/crictl (0.11s)
        --- PASS: TestISOImage/Binaries/git (0.11s)
        --- PASS: TestISOImage/Binaries/rsync (0.07s)
        --- PASS: TestISOImage/Binaries/docker (0.08s)
        --- PASS: TestISOImage/Binaries/curl (0.08s)
    --- PASS: TestISOImage/PersistentMounts (0.00s)
        --- PASS: TestISOImage/PersistentMounts//var/lib/toolbox (0.08s)
        --- PASS: TestISOImage/PersistentMounts//var/lib/boot2docker (0.09s)
        --- PASS: TestISOImage/PersistentMounts//var/lib/cni (0.09s)
        --- PASS: TestISOImage/PersistentMounts//var/lib/minikube (0.10s)
        --- PASS: TestISOImage/PersistentMounts//data (0.10s)
        --- PASS: TestISOImage/PersistentMounts//var/lib/docker (0.11s)
        --- PASS: TestISOImage/PersistentMounts//var/lib/kubelet (0.07s)
PASS
Tests completed in 5.7657725s (result code 0)
ok  	k8s.io/minikube/test/integration	6.974s

nirs · 2025-10-25T21:36:42Z

/ok-to-test

nirs · 2025-10-25T21:45:17Z

/cc @medyagh

The test used `go:build iso` so it was not included in the integration tests. Change to `go:build integration` so we test in the CI. Rename the file and the test name to make it more clear that this test is about the iso image. Skip the test for non vm-driver, since this tests is about the iso image. We may need to add a similar test or adapt this test so it can be used also with the kicbase image. This test will be useful to validate kubernetes#21800, avoiding regressions such as kubernetes#21788.

Virtualbox binaries are not built in the arm64 iso. Test them only on other architectures. Example run: % go test -v ./test/integration -run TestISO -tags integration --- PASS: TestISOImage (17.74s) --- PASS: TestISOImage/Setup (17.31s) --- PASS: TestISOImage/Binaries (0.00s) --- PASS: TestISOImage/Binaries/docker (0.08s) --- PASS: TestISOImage/Binaries/podman (0.09s) --- PASS: TestISOImage/Binaries/iptables (0.10s) --- PASS: TestISOImage/Binaries/git (0.10s) --- PASS: TestISOImage/Binaries/curl (0.11s) --- PASS: TestISOImage/Binaries/crictl (0.11s) --- PASS: TestISOImage/Binaries/wget (0.06s) --- PASS: TestISOImage/Binaries/socat (0.07s) --- PASS: TestISOImage/Binaries/rsync (0.06s) --- PASS: TestISOImage/PersistentMounts (0.00s) --- PASS: TestISOImage/PersistentMounts//var/lib/boot2docker (0.08s) --- PASS: TestISOImage/PersistentMounts//var/lib/cni (0.09s) --- PASS: TestISOImage/PersistentMounts//var/lib/toolbox (0.09s) --- PASS: TestISOImage/PersistentMounts//var/lib/minikube (0.10s) --- PASS: TestISOImage/PersistentMounts//var/lib/docker (0.10s) --- PASS: TestISOImage/PersistentMounts//data (0.11s) --- PASS: TestISOImage/PersistentMounts//var/lib/kubelet (0.06s)

nirs · 2025-10-26T15:08:47Z

Test results on top of #21800

successful checks

deploy/netlify
deploy/netlify — Deploy Preview ready!
Docker_Linux
Docker_Linux — Jenkins: completed with success in 47.83 minutes.
Docker_Linux_containerd
Docker_Linux_containerd — Jenkins: completed with success in 37.56 minutes.
Docker_Linux_docker_arm64
Docker_Linux_docker_arm64 — Jenkins: completed with success in 69.65 minutes.
EasyCLA
EasyCLA — EasyCLA check passed. You are authorized to contribute.
Required
Functional Test / baremetal-docker-ubuntu-22.04-x86_64 (pull_request)
Functional Test / baremetal-docker-ubuntu-22.04-x86_64 (pull_request)Successful in 5m
Functional Test / build-test-binaries (pull_request)
Functional Test / build-test-binaries (pull_request)Successful in 4m
Functional Test / docker-containerd-rootless-ubuntu-22.04-x86_64 (pull_request)
Functional Test / docker-containerd-rootless-ubuntu-22.04-x86_64 (pull_request)Successful in 9m
Functional Test / docker-containerd-ubuntu-22.04-x86_64 (pull_request)
Functional Test / docker-containerd-ubuntu-22.04-x86_64 (pull_request)Successful in 10m
Functional Test / docker-docker-ubuntu22.04-x86_64 (pull_request)
Functional Test / docker-docker-ubuntu22.04-x86_64 (pull_request)Successful in 14m
Functional Test / podman-docker-ubuntu-24.04-x86_64 (pull_request)
Functional Test / podman-docker-ubuntu-24.04-x86_64 (pull_request)Successful in 13m
Functional_Verified / build_minikube (pull_request)
Functional_Verified / build_minikube (pull_request)Successful in 5m
Functional_Verified / functional_docker_ubuntu_arm64 (pull_request)
Functional_Verified / functional_docker_ubuntu_arm64 (pull_request)Successful in 5m
Functional_Verified / upload_all_reports (pull_request)
Functional_Verified / upload_all_reports (pull_request)Successful in 15s
Jenkins Cross Build
Jenkins Cross Build — Build finished. No test results found.
KVM_Linux
KVM_Linux — Jenkins: completed with success in 64.43 minutes.
KVM_Linux_containerd
KVM_Linux_containerd — Jenkins: completed with success in 67.26 minutes.
Lint / Boilerplate (pull_request)
Lint / Boilerplate (pull_request)Successful in 22s
Lint / Gomodtidy (pull_request)
Lint / Gomodtidy (pull_request)Successful in 22s
Lint / Lint- (pull_request)
Lint / Lint- (pull_request)Successful in 4m
none_Linux
none_Linux — Jenkins: completed with success in 27.65 minutes.
pull-minikube-build
pull-minikube-build — Job succeeded. BaseSHA:5ca02f9f576d2f3fd8ffc33bc1ce011b281a7e09
Required
Smoke Test / Boot (docker, ubuntu-24.04) (pull_request)
Smoke Test / Boot (docker, ubuntu-24.04) (pull_request)Successful in 1m
Smoke Test / Boot (vfkit, macos-13, --network vmnet-shared) (pull_request)
Smoke Test / Boot (vfkit, macos-13, --network vmnet-shared) (pull_request)Successful in 7m
Unit Test / unit_test (macos-15) (pull_request)
Unit Test / unit_test (macos-15) (pull_request)Successful in 2m
Unit Test / unit_test (ubuntu-22.04) (pull_request)
Unit Test / unit_test (ubuntu-22.04) (pull_request)Successful in 3m

nirs · 2025-10-27T10:42:46Z

@medyagh should be ready for merge

/assign @medyagh

medyagh · 2025-10-27T17:38:43Z

test/integration/iso_test.go

-// TestGuestEnvironment verifies files and packages installed inside minikube ISO/Base image
-func TestGuestEnvironment(t *testing.T) {
+// TestISOImage verifies files and packages installed inside minikube ISO/Base image
+func TestISOImage(t *testing.T) {


Thank you for finding out this test never ran !!!

How about we make this a SubTest of No-Kubernetes Test ? so we spin up one No-Kuberentes VM and dont spin up any more new VMs

Using --no-kubernetes makes sense. I'll check this out.

I don't think we should make this a sub test of TestNoKubernetes.

The test is complicated enough as is:

// Serial tests t.Run("serial", func(t *testing.T) { tests := []struct { name string validator validateFunc }{ {"StartNoK8sWithVersion", validateStartNoK8sWithVersion}, {"StartWithK8s", validateStartWithK8S}, {"StartWithStopK8s", validateStartWithStopK8s}, {"Start", validateStartNoK8S}, {"VerifyK8sNotRunning", validateK8SNotRunning}, {"ProfileList", validateProfileListNoK8S}, {"Stop", validateStopNoK8S}, {"StartNoArgs", validateStartNoArgs}, {"VerifyK8sNotRunningSecond", validateK8SNotRunning}, } for _, tc := range tests { tc := tc if ctx.Err() == context.DeadlineExceeded { t.Fatalf("Unable to run more tests (deadline exceeded)") } t.Run(tc.name, func(t *testing.T) { tc.validator(ctx, t, profile) if t.Failed() && *postMortemLogs { PostMortemLogs(t, profile) } }) } })

Adding the test as sub test will make it impossible to run the test separately like we can do with the current PR. It may not run the test if a previous unrelated test failed, and it will make it harder to maintain the tests.

What we can as should do is to start the cluster with --no-kubernetes in the iso test since we test the iso contents, and kubernetes is not required of this. This will make the test much quicker locally.

Using --no-kubernetes now.

We don't need to start kubernetes for testing the ISO. On macOS the test is 2.45 times faster (from 17.137 seconds to 6.974 seconds). Before: % go test -v ./test/integration -run TestISO -tags integration -count 1 ... --- PASS: TestISOImage (16.27s) --- PASS: TestISOImage/Setup (15.76s) --- PASS: TestISOImage/Binaries (0.00s) --- PASS: TestISOImage/Binaries/crictl (0.08s) --- PASS: TestISOImage/Binaries/podman (0.09s) --- PASS: TestISOImage/Binaries/rsync (0.11s) --- PASS: TestISOImage/Binaries/docker (0.11s) --- PASS: TestISOImage/Binaries/iptables (0.11s) --- PASS: TestISOImage/Binaries/git (0.12s) --- PASS: TestISOImage/Binaries/socat (0.08s) --- PASS: TestISOImage/Binaries/curl (0.08s) --- PASS: TestISOImage/Binaries/wget (0.08s) --- PASS: TestISOImage/PersistentMounts (0.00s) --- PASS: TestISOImage/PersistentMounts//var/lib/toolbox (0.08s) --- PASS: TestISOImage/PersistentMounts//var/lib/boot2docker (0.09s) --- PASS: TestISOImage/PersistentMounts//var/lib/cni (0.09s) --- PASS: TestISOImage/PersistentMounts//data (0.10s) --- PASS: TestISOImage/PersistentMounts//var/lib/kubelet (0.10s) --- PASS: TestISOImage/PersistentMounts//var/lib/minikube (0.11s) --- PASS: TestISOImage/PersistentMounts//var/lib/docker (0.07s) PASS Tests completed in 16.268875292s (result code 0) ok k8s.io/minikube/test/integration 17.137s After: % go test -v ./test/integration -run TestISO -tags integration -count 1 ... --- PASS: TestISOImage (5.77s) --- PASS: TestISOImage/Setup (5.30s) --- PASS: TestISOImage/Binaries (0.00s) --- PASS: TestISOImage/Binaries/wget (0.09s) --- PASS: TestISOImage/Binaries/socat (0.09s) --- PASS: TestISOImage/Binaries/iptables (0.10s) --- PASS: TestISOImage/Binaries/podman (0.10s) --- PASS: TestISOImage/Binaries/crictl (0.11s) --- PASS: TestISOImage/Binaries/git (0.11s) --- PASS: TestISOImage/Binaries/rsync (0.07s) --- PASS: TestISOImage/Binaries/docker (0.08s) --- PASS: TestISOImage/Binaries/curl (0.08s) --- PASS: TestISOImage/PersistentMounts (0.00s) --- PASS: TestISOImage/PersistentMounts//var/lib/toolbox (0.08s) --- PASS: TestISOImage/PersistentMounts//var/lib/boot2docker (0.09s) --- PASS: TestISOImage/PersistentMounts//var/lib/cni (0.09s) --- PASS: TestISOImage/PersistentMounts//var/lib/minikube (0.10s) --- PASS: TestISOImage/PersistentMounts//data (0.10s) --- PASS: TestISOImage/PersistentMounts//var/lib/docker (0.11s) --- PASS: TestISOImage/PersistentMounts//var/lib/kubelet (0.07s) PASS Tests completed in 5.7657725s (result code 0) ok k8s.io/minikube/test/integration 6.974s

k8s-ci-robot · 2025-10-27T18:38:05Z

@nirs: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
integration-kvm-docker-linux-x86-64	`3dc60e2`	link	true	`/test integration-kvm-docker-linux-x86-64`
integration-docker-crio-linux-x86-64	`3dc60e2`	link	true	`/test integration-docker-crio-linux-x86-64`
integration-docker-docker-linux-x86-64	`3dc60e2`	link	true	`/test integration-docker-docker-linux-x86-64`
integration-kvm-crio-linux-x86-64	`3dc60e2`	link	true	`/test integration-kvm-crio-linux-x86-64`
integration-none-docker-linux-x86-64	`3dc60e2`	link	true	`/test integration-none-docker-linux-x86-64`
integration-kvm-containerd-linux-x86-64	`3dc60e2`	link	true	`/test integration-kvm-containerd-linux-x86-64`
integration-docker-containerd-linux-x86-64	`3dc60e2`	link	true	`/test integration-docker-containerd-linux-x86-64`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

minikube-pr-bot · 2025-10-27T19:24:19Z

kvm2 driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21801 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 43.6s    │ 44.9s                  │
│ enable ingress │ 16.0s    │ 18.6s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube (PR 21801) ingress: 15.9s 17.3s 20.9s 19.4s 19.8s
Times for minikube ingress: 16.3s 15.8s 15.8s 15.9s 16.3s

Times for minikube (PR 21801) start: 46.0s 45.3s 44.4s 46.1s 42.6s
Times for minikube start: 40.4s 44.2s 43.8s 46.5s 43.4s

docker driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21801 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 23.2s    │ 24.0s                  │
│ enable ingress │ 11.3s    │ 11.7s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube (PR 21801) start: 24.7s 26.7s 23.7s 22.9s 22.1s
Times for minikube start: 23.6s 25.1s 22.7s 22.8s 22.0s

Times for minikube (PR 21801) ingress: 10.7s 12.7s 10.7s 12.7s 11.7s
Times for minikube ingress: 12.7s 10.7s 10.7s 10.7s 11.7s

docker driver with containerd runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21801 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 20.9s    │ 21.2s                  │
│ enable ingress │ 21.6s    │ 21.6s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 20.9s 21.3s 20.3s 22.2s 19.8s
Times for minikube (PR 21801) start: 21.5s 22.5s 20.9s 21.1s 20.2s

Times for minikube (PR 21801) ingress: 20.2s 21.2s 22.2s 23.2s 21.1s
Times for minikube ingress: 23.2s 20.2s 22.2s 20.2s 22.2s

nirs · 2025-10-27T19:43:29Z

Example run in Fedora Linux VM:

$ go test -v ./test/integration -run TestISO -tags integration -count 1 -minikube-start-args '--driver=kvm'
...
--- PASS: TestISOImage (27.22s)
    --- PASS: TestISOImage/Setup (24.54s)
    --- PASS: TestISOImage/Binaries (0.00s)
        --- PASS: TestISOImage/Binaries/VBoxService (0.50s)
        --- PASS: TestISOImage/Binaries/wget (0.51s)
        --- PASS: TestISOImage/Binaries/rsync (0.53s)
        --- PASS: TestISOImage/Binaries/crictl (0.55s)
        --- PASS: TestISOImage/Binaries/VBoxControl (0.56s)
        --- PASS: TestISOImage/Binaries/podman (0.37s)
        --- PASS: TestISOImage/Binaries/socat (0.43s)
        --- PASS: TestISOImage/Binaries/iptables (0.42s)
        --- PASS: TestISOImage/Binaries/git (0.44s)
        --- PASS: TestISOImage/Binaries/docker (0.45s)
        --- PASS: TestISOImage/Binaries/curl (0.27s)
    --- PASS: TestISOImage/PersistentMounts (0.00s)
        --- PASS: TestISOImage/PersistentMounts//var/lib/cni (0.45s)
        --- PASS: TestISOImage/PersistentMounts//var/lib/minikube (0.46s)
        --- PASS: TestISOImage/PersistentMounts//var/lib/docker (0.50s)
        --- PASS: TestISOImage/PersistentMounts//data (0.51s)
        --- PASS: TestISOImage/PersistentMounts//var/lib/kubelet (0.53s)
        --- PASS: TestISOImage/PersistentMounts//var/lib/boot2docker (0.36s)
        --- PASS: TestISOImage/PersistentMounts//var/lib/toolbox (0.35s)
PASS
Tests completed in 27.216404548s (result code 0)
ok  	k8s.io/minikube/test/integration	27.289s

$ go test -v ./test/integration -run TestISO -tags integration -count 1 -minikube-start-args '--driver=docker'
Found 10 cores, limiting parallelism with --test.parallel=5
=== RUN   TestISOImage
    iso_test.go:34: This test requires a VM driver
--- SKIP: TestISOImage (0.00s)
PASS
Tests completed in 180.348µs (result code 0)
ok  	k8s.io/minikube/test/integration	0.067s

medyagh · 2025-10-27T20:41:08Z

test/integration/iso_test.go

 	// Run as a group so that our defer doesn't happen as tests are runnings
 	t.Run("Binaries", func(t *testing.T) {
-		for _, pkg := range []string{"git", "rsync", "curl", "wget", "socat", "iptables", "VBoxControl", "VBoxService", "crictl", "podman", "docker"} {
+		binaries := []string{


optional but we can check for other things too, we put a File that has the minikube version in there

example:

$ cat /version.json
{"iso_version": "v1.37.0-1758198818-20370", "kicbase_version": "v0.0.48", "minikube_version": "v1.37.0", "commit": "a4f96d0469d67330691be52a99ff1f91e31ba77f"}

Good idea! open #21815 as good first issue.

medyagh · 2025-10-27T20:55:22Z

/lgtm

k8s-ci-robot · 2025-10-27T20:55:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, nirs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [medyagh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

minikube-pr-bot · 2025-10-27T22:56:15Z

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment	Test Name	Flake Rate
Docker_Windows (2 failed)	TestErrorSpam/setup(gopogh)	Unknown
Docker_Windows (2 failed)	TestStartStop/group/newest-cni/serial/Pause(gopogh)	Unknown

Besides the following environments also have failed tests:

KVM_Linux_crio: 14 failed (gopogh)
Docker_Linux_crio_arm64: 36 failed (gopogh)
Docker_Linux_crio: 37 failed (gopogh)

To see the flake rates of all tests by environment, click here.

k8s-ci-robot requested review from ComradeProgrammer and prezha October 25, 2025 21:34

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 25, 2025

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Oct 25, 2025

k8s-ci-robot requested a review from medyagh October 25, 2025 21:45

This comment has been minimized.

Sign in to view

nirs mentioned this pull request Oct 26, 2025

iso: Enable BTF debug symbols to support eBPF tooling #21800

Merged

nirs force-pushed the iso-test branch from 4dab035 to e76bf7b Compare October 26, 2025 11:58

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 26, 2025

This comment has been minimized.

Sign in to view

nirs added 2 commits October 26, 2025 17:05

nirs force-pushed the iso-test branch from e76bf7b to 5bd82cb Compare October 26, 2025 15:09

This comment has been minimized.

Sign in to view

k8s-ci-robot assigned medyagh Oct 27, 2025

medyagh requested changes Oct 27, 2025

View reviewed changes

nirs requested a review from medyagh October 27, 2025 18:41

medyagh requested changes Oct 27, 2025

View reviewed changes

nirs mentioned this pull request Oct 27, 2025

test: Verify and log iso version in iso_test.go #21815

Open

medyagh merged commit b71c807 into kubernetes:master Oct 27, 2025
29 of 48 checks passed

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 27, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 27, 2025

test: Fix guest_env_test.go #21801

test: Fix guest_env_test.go #21801

Uh oh!

Conversation

nirs commented Oct 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example run

Uh oh!

nirs commented Oct 25, 2025

Uh oh!

nirs commented Oct 25, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nirs commented Oct 26, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nirs commented Oct 27, 2025

Uh oh!

medyagh Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

nirs Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

nirs Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

nirs Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Oct 27, 2025

Uh oh!

minikube-pr-bot commented Oct 27, 2025

Uh oh!

nirs commented Oct 27, 2025

Uh oh!

medyagh Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

nirs Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

medyagh commented Oct 27, 2025

Uh oh!

Uh oh!

k8s-ci-robot commented Oct 27, 2025

Uh oh!

minikube-pr-bot commented Oct 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nirs commented Oct 25, 2025 •

edited

Loading